[Fleet] Extend OTel exporter configuration by criamico · Pull Request #259308 · elastic/kibana

criamico · 2026-03-24T11:17:24Z

Summary

Extend OTel exporter configuration: The existing exporter generation only mapped hosts from an Elasticsearch output. All other config (SSL, proxy, TLS, custom params) was ignored.

Adds a beatsauth/<outputID> extension to every OTel policy that uses an ES output. This extension carries SSL fields and proxy settings from the Fleet output config.
Merges the new otel_exporter_config_yaml field into the exporter config.
preset is intentionally ignored for now (the exporter uses its own defaults).
Adds an "OpenTelemetry Exporter" panel (ES outputs only) with an "Advanced YAML Configuration" editor. Content is saved as otel_exporter_config_yaml and merged into the OTel exporter at policy generation time.

Testing

Create an Elasticsearch output with SSL settings in Settings → Outputs → Add output:

Type: Elasticsearch
Hosts: https://your-es-host:9200
CA Trusted Fingerprint: any hex string, e.g. abc123def456
(Optionally add SSL certificate/key under Advanced)

Create an agent policy with an OTel integration in Agent Policies → Create policy
Assign the output you just created to the policy
Add an OTel-based integration - I tested with filelog_otel-0.2.0 but any otel input integration (not content only) will do
Verify the generated agent policy. In the response, look for the top-level extensions, exporters, and service keys. You should see:

extensions:
  beatsauth/<output-id>:          # e.g. beatsauth/default or beatsauth/my-output-id
    ssl:
      ca_trusted_fingerprint: abc123def456
      # ... any other SSL fields you configured

exporters:
  elasticsearch/<output-id>:
    endpoints:
      - https://your-es-host:9200
    auth:
      authenticator: beatsauth/<output-id>   # ← new field

service:
  extensions:
    - beatsauth/<output-id>                  # ← registered in service

Test with a proxy - Settings → Proxies → Add proxy → set a URL
Edit your output and assign the proxy. Re-fetch the full agent policy — the beatsauth extension should now also include:

beatsauth/<output-id>:
  proxy_url: http://localhost:3128
  ssl:
    ca_trusted_fingerprint: abc123def456

Test with a yaml custom parameter - add a yaml parameter under Advanced YAML Configuration
This parameter should now be visible in the generated policy

Screenshots

Checklist

Check the PR satisfies following conditions.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios
If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The release_note:breaking label should be applied in these situations.
Flaky Test Runner was used on any tests changed
The PR description includes the appropriate Release Notes section, and the correct release_note:* label is applied per the guidelines
Review the backport guidelines and apply applicable backport:* labels.

Merge main

criamico · 2026-03-25T14:55:14Z

@elasticmachine merge upstream

criamico · 2026-03-25T16:07:25Z

Example of policy generated after this change:

id: e60a406d-7f8e-4afc-bfc6-b399f55090f9
revision: 6
outputs:
  a32c37a7-900b-4440-8c06-a6514e095098:
    type: elasticsearch
    hosts:
      - http://test-host:80
    secrets:
      ssl:
        key:
          id: 6ZAvH50BFEDfHFivzCrf
    ssl:
      certificate: certificate
      certificate_authorities: []
      key: '-----BEGIN PRIVATE KEY-----LEAK_ME_PROXY_KEY-----END PRIVATE KEY-----'
    proxy_url: https://proxy.internal.example:8443
    proxy_headers:
      Authorization: Bearer PROXY_SECRET
    preset: balanced
fleet:
  hosts:
    - https://host.docker.internal:8220
output_permissions:
  a32c37a7-900b-4440-8c06-a6514e095098:
    _elastic_agent_monitoring:
      indices:
        - names:
            - logs-elastic_agent.apm_server-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.apm_server-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.auditbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.auditbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.cloud_defend-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.cloudbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.cloudbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.elastic_agent-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.endpoint_security-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.endpoint_security-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.filebeat_input-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.filebeat_input-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.filebeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.filebeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.fleet_server-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.fleet_server-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.heartbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.heartbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.metricbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.metricbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.osquerybeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.osquerybeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.packetbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.packetbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.pf_elastic_collector-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.pf_elastic_symbolizer-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.pf_host_agent-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.status_change-default
          privileges:
            - auto_configure
            - create_doc
    _elastic_agent_checks:
      cluster:
        - monitor
    1a458fca-7c0f-4cf0-bd2c-291ff8844f1c:
      indices:
        - names:
            - logs-*-*
          privileges:
            - auto_configure
            - create_doc
agent:
  download:
    sourceURI: https://artifacts.elastic.co/downloads/
    ssl:
      key: '-----BEGIN PRIVATE KEY-----LEAK_ME_PROXY_KEY-----END PRIVATE KEY-----'
    proxy_url: https://proxy.internal.example:8443
    proxy_headers:
      Authorization: Bearer PROXY_SECRET
  monitoring:
    enabled: true
    use_output: a32c37a7-900b-4440-8c06-a6514e095098
    logs: true
    metrics: true
    traces: true
    namespace: default
  features: {}
  protection:
    enabled: false
    uninstall_token_hash: ***
    signing_key: ***
inputs: []
signed:
 ...
receivers:
  filelog/otelcol-filelogreceiver-1a458fca-7c0f-4cf0-bd2c-291ff8844f1c-otelcol-filelog_otel-filelogreceiver-1a458fca-7c0f-4cf0-bd2c-291ff8844f1c:
    include:
      - aaaa
    start_at: end
    max_concurrent_files: 1024
    include_file_name: true
    include_file_path: false
    include_file_name_resolved: false
    include_file_path_resolved: false
    include_file_owner_name: false
    include_file_owner_group_name: false
    include_file_record_number: false
    include_file_record_offset: false
    encoding: utf-8
    preserve_leading_whitespaces: false
    preserve_trailing_whitespaces: false
    fingerprint_size: 1kb
    initial_buffer_size: 16KiB
    max_log_size: 1MiB
    poll_interval: 200ms
    force_flush_period: 500ms
service:
  pipelines:
    logs/otelcol-filelogreceiver-1a458fca-7c0f-4cf0-bd2c-291ff8844f1c-otelcol-filelog_otel-filelogreceiver-1a458fca-7c0f-4cf0-bd2c-291ff8844f1c:
      receivers:
        - >-
          filelog/otelcol-filelogreceiver-1a458fca-7c0f-4cf0-bd2c-291ff8844f1c-otelcol-filelog_otel-filelogreceiver-1a458fca-7c0f-4cf0-bd2c-291ff8844f1c
      processors:
        - >-
          transform/otelcol-filelogreceiver-1a458fca-7c0f-4cf0-bd2c-291ff8844f1c-otelcol-filelog_otel-filelogreceiver-1a458fca-7c0f-4cf0-bd2c-291ff8844f1c-routing
      exporters:
        - forward
    logs:
      receivers:
        - forward
      exporters:
        - elasticsearch/a32c37a7-900b-4440-8c06-a6514e095098
  extensions:
    - beatsauth/a32c37a7-900b-4440-8c06-a6514e095098
processors:
  transform/otelcol-filelogreceiver-1a458fca-7c0f-4cf0-bd2c-291ff8844f1c-otelcol-filelog_otel-filelogreceiver-1a458fca-7c0f-4cf0-bd2c-291ff8844f1c-routing:
    log_statements:
      - context: log
        statements:
          - set(attributes["data_stream.type"], "logs")
          - set(attributes["data_stream.dataset"], "filelogreceiver")
          - set(attributes["data_stream.namespace"], "default")
connectors:
  forward: {}
extensions:
  beatsauth/a32c37a7-900b-4440-8c06-a6514e095098:
    ssl:
      certificate: certificate
    proxy_url: https://proxy.internal.example:8443
    proxy_headers:
      Authorization: Bearer PROXY_SECRET
exporters:
  elasticsearch/a32c37a7-900b-4440-8c06-a6514e095098:
    flush_interval: 10s
    endpoints:
      - http://test-host:80
    auth:
      authenticator: beatsauth/a32c37a7-900b-4440-8c06-a6514e095098
secret_references:
  - id: 6ZAvH50BFEDfHFivzCrf
namespaces:
  - default

elasticmachine · 2026-03-25T16:07:43Z

Pinging @elastic/fleet (Team:Fleet)

juliaElastic · 2026-03-26T13:48:22Z

+  if (output.ssl?.certificate_authorities?.length)
+    ssl.certificate_authorities = output.ssl.certificate_authorities;
+  if (output.ssl?.certificate) ssl.certificate = output.ssl.certificate;
+  if (output.ssl?.key) ssl.key = output.ssl.key;


should we handle secrets here?

good catch, I missed it

juliaElastic · 2026-03-26T13:52:10Z

-        [`elasticsearch/${outputID}`]: {
-          endpoints: dataOutput.hosts,
+        extensions: {
+          [beatsauthID]: buildBeatsauthConfig(dataOutput, proxy),


when buildBeatsauthConfig returns {}, should we still add beatsauth to extensions and exporters?

I think we are still adding a beatsuath extension and exporter if the config is empty

Fixed in 4451e00

criamico · 2026-03-26T16:16:26Z

@elasticmachine merge upstream

…tion_tests/ci_checks

juliaElastic

LGTM

criamico · 2026-04-02T08:58:04Z

@elasticmachine merge upstream

…tion_tests/ci_checks

criamico · 2026-04-02T14:29:45Z

@jloleysens there is a failure that keeps happening in the check API contract check: https://buildkite.com/elastic/kibana-pull-request/builds/421398#019d4e7c-00ec-48ad-b6bb-dd0b2f0ad693

I think it's not regenerating the OAS doc, how can I fix it?

…tion_tests/ci_checks

kibanamachine · 2026-04-07T15:48:04Z

API Contract Breaking Changes — Terraform Provider Impact

cc @elastic/fleet

The following breaking change(s) affect APIs consumed by the Elastic Terraform Provider.

Endpoint	Terraform Resource	Reason	Owners
`/api/fleet/outputs/{outputId}` `PUT`	elasticstack_fleet_output	removed 'subschema #1, subschema #2, subschema #3, subschema #4' from the request body 'anyOf' list	@elastic/fleet

What to do

Fix the breaking change if it was unintentional.
If intentional, add an approved entry to packages/kbn-api-contracts/allowlist.json and coordinate with @elastic/terraform-provider.

See the @kbn/api-contracts README for details on the allowlist schema and workflow.

criamico · 2026-04-08T09:22:29Z

@elasticmachine merge upstream

criamico · 2026-04-08T10:43:38Z

@elasticmachine merge upstream

elasticmachine · 2026-04-08T11:58:55Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: a559e3b

Failed CI Steps

Fleet Cypress Tests #1

Test Failures

[job] [logs] Fleet Cypress Tests #1 / View agents list Bulk actions should navigate to Maintenance and diagnostics submenu
[job] [logs] Fleet Cypress Tests #1 / View agents list Bulk actions should navigate to Security and removal submenu and show unenroll option
[job] [logs] Fleet Cypress Tests #1 / View agents list Bulk actions should show hierarchical menu with submenus

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`fleet`	2.3MB	2.3MB	+1.6KB

History

💔 Build #423822 failed af3c2a9
💔 Build #423334 failed 0fa70a8
💔 Build #423082 failed 8279ec6
💔 Build #422940 failed 500aa17
💔 Build #421398 failed 5ad8e1e
💔 Build #421121 failed b29c954

cc @criamico

criamico added 3 commits March 23, 2026 11:28

[Fleet] Extend OTel exporter configuration

a00f884

Merge branch 'main' into 255019_extend_otel_exporter_config

5a8b168

Merge main

Add advanced section

9e8243d

criamico changed the title ~~255019 extend otel exporter config~~ [Fleet] Extend OTel exporter configuration Mar 25, 2026

Improvements to UI

d113d4b

elasticmachine and others added 2 commits March 25, 2026 07:55

Merge branch 'main' into 255019_extend_otel_exporter_config

58b68bf

update test

f802353

macroscopeapp Bot reviewed Mar 25, 2026

View reviewed changes

Comment thread ...eet/public/applications/fleet/sections/settings/components/edit_output_flyout/index.test.tsx Outdated

Fix test and handle malformed yaml

47b49c8

criamico self-assigned this Mar 25, 2026

criamico added Team:Fleet Team label for Observability Data Collection Fleet team v9.4.0 release_note:skip Skip the PR/issue when compiling release notes backport:skip This PR does not require backporting labels Mar 25, 2026

criamico marked this pull request as ready for review March 25, 2026 16:07

criamico requested review from a team as code owners March 25, 2026 16:07

criamico requested a review from rgodfrey-elastic March 25, 2026 16:07

juliaElastic reviewed Mar 26, 2026

View reviewed changes

Comment thread x-pack/platform/plugins/shared/fleet/server/saved_objects/index.ts

juliaElastic reviewed Mar 26, 2026

View reviewed changes

Comment thread x-pack/platform/plugins/shared/fleet/server/services/agent_policies/otel_collector.ts

juliaElastic reviewed Mar 26, 2026

View reviewed changes

Address code review comments

94e84cd

elasticmachine and others added 2 commits March 26, 2026 09:16

Merge branch 'main' into 255019_extend_otel_exporter_config

024ddc5

Changes from node scripts/check_mappings_update --fix

99235d1

elastic-vault-github-plugin-prod Bot requested a review from a team as a code owner March 26, 2026 16:36

elasticmachine and others added 5 commits April 1, 2026 00:48

Merge branch 'main' into 255019_extend_otel_exporter_config

e4e43e6

Update schema to fix test

992835d

Changes from node scripts/jest_integration -u src/core/server/integra…

59c611c

…tion_tests/ci_checks

Fix schema again

01d6cab

Changes from node scripts/jest_integration -u src/core/server/integra…

5a49dbf

…tion_tests/ci_checks

juliaElastic approved these changes Apr 2, 2026

View reviewed changes

elasticmachine and others added 3 commits April 2, 2026 01:58

Merge branch 'main' into 255019_extend_otel_exporter_config

b29c954

Simplify outputs schema

6e666f5

Changes from node scripts/jest_integration -u src/core/server/integra…

5ad8e1e

…tion_tests/ci_checks

rgodfrey-elastic approved these changes Apr 2, 2026

View reviewed changes

Merge main and fix conflicts

500aa17

macroscopeapp Bot reviewed Apr 7, 2026

View reviewed changes

Comment thread x-pack/platform/plugins/shared/fleet/server/saved_objects/index.ts

criamico and others added 4 commits April 7, 2026 11:27

Address Macroscope app review

b275adb

Changes from node scripts/jest_integration -u src/core/server/integra…

8279ec6

…tion_tests/ci_checks

Fix oas docs

5d62d10

Changes from make api-docs

0fa70a8

Add allowlist for API contract check

ceacab8

Merge branch 'main' into 255019_extend_otel_exporter_config

af3c2a9

Merge branch 'main' into 255019_extend_otel_exporter_config

a559e3b

criamico merged commit d8598b8 into elastic:main Apr 8, 2026
21 checks passed

criamico mentioned this pull request Apr 8, 2026

Fix fleet output OAS regressions: SSL type explosion and Kafka union wrappers #260842

Merged

3 tasks

cmacknz mentioned this pull request Apr 13, 2026

[beats receivers] beatsauth extensions specified by users are not merged correctly with automatically generated ones elastic/elastic-agent#13610

Closed

This was referenced Apr 15, 2026

Policy tests fail on 9.4.0+ due to new beatsauth OTel fields injected by Fleet elastic/elastic-package#3463

Closed

Fix policy tests failing on 9.4.0+ due to beatsauth OTel fields injected by Fleet elastic/elastic-package#3467

Merged

Conversation

criamico commented Mar 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Testing

Checklist

Uh oh!

criamico commented Mar 25, 2026

Uh oh!

Uh oh!

criamico commented Mar 25, 2026

Uh oh!

elasticmachine commented Mar 25, 2026

Uh oh!

Uh oh!

Uh oh!

juliaElastic Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

criamico Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

juliaElastic Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

juliaElastic Mar 30, 2026

Choose a reason for hiding this comment

Uh oh!

criamico Mar 30, 2026

Choose a reason for hiding this comment

Uh oh!

criamico commented Mar 26, 2026

Uh oh!

juliaElastic left a comment

Choose a reason for hiding this comment

Uh oh!

criamico commented Apr 2, 2026

Uh oh!

criamico commented Apr 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

kibanamachine commented Apr 7, 2026

API Contract Breaking Changes — Terraform Provider Impact

What to do

Uh oh!

criamico commented Apr 8, 2026

Uh oh!

criamico commented Apr 8, 2026

Uh oh!

elasticmachine commented Apr 8, 2026

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

Metrics [docs]

Async chunks

History

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

criamico commented Mar 24, 2026 •

edited

Loading

criamico commented Apr 2, 2026 •

edited

Loading